Centurion_SignedDataWorker_Issue
Incident Report: Centurion SignedDataWorker Issue
Date: 2024-06-04
Time: 22:15 (GMT+2)
Duration: 13 minutes
Descriptionβ
Unexpected downtime and delay in data updates for ETH/USD on Mantle, Blast, Kava, Taiko, and BTC/USD on Kava and Taiko.
Root Causeβ
The issue was caused by the signed API not returning data due to an error in the response object being undefined during the sync history process.
Impactβ
Event collection was unavailable for the specified chains, leading to a 13-minute data gap on the Centurion platform.
Timelineβ
- 22:15 - First noticed the issue.
- 22:17 - Initial diagnosis began.
- 22:33 - Started working on a fix.
- 14:44 - Issue resolved.
Actions Takenβ
- Verification of Endpoints: Verified the signed API endpoints for various pairs.
- Identification of Error: Identified that the response object was undefined due to a TypeError in the sync history process.
- RPC Providers Check: Manually checked and validated RPC providers for correct gas prices.
- Airseeker Configuration Update: Updated Airseeker configuration to include reliable RPC providers, specifically Ankr and https://rpc.taiko.tools/.
- Local Deployment: Deployed the updated configuration locally to quickly resolve the issue and later deployed to one region (eu-north-1).
- Monitoring Enhancement: Created an issue to add better detection for undefined responses in the sync history process.
Related Images/Logsβ
- Escalation link.
Incident Reviewer(s)β
- Arda
- Warren
- Ali
- Bedirhan
- Emanuel TesaΕ
- Mertcan Karik
- Andrew Prasaath
- Abdel